Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add data node migration wizard #17768

Merged
merged 98 commits into from
Feb 22, 2024
Merged

Add data node migration wizard #17768

merged 98 commits into from
Feb 22, 2024

Conversation

ousmaneo
Copy link
Contributor

@ousmaneo ousmaneo commented Dec 21, 2023

These changes add UI for the Datanode migration API. The step in the migration wizard uses the state machine API.

image (1)

fix #17703

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Refactoring (non-breaking change)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.

@ousmaneo ousmaneo force-pushed the datanode-migration-ui branch from 8af0564 to 2f17c7a Compare January 16, 2024 13:56
Copy link
Contributor

@grotlue grotlue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still need to test

  • When OpenSearch is not stopped
  • Certificate Upload
  • Certificate renewal policy tab

I am already going to post all findings and thoughts that I have so far. In general, there are a lot of typos, extra spaces, misspellings of "terms" like OpenSearch, Data Node, etc.
I tried to point out all that I found but please have another look. I think spelling mistakes like that don't make a feature appear mature.

Code-wise it looks good, just some nit-picks, nothing crucial.

Wizard in general

  • Typo Certificate autority, should be Certificate authority
  • I can click on non-active steps (previous and next) but nothing happens. Confusingly, the clicked step stays in a visual focused state
  • If I go from Certificate authority to Migration Steps there is no way to get back again
  • I cannot see an option to cancel the migration process and/or start from step 1 again. After reloading I am still stuck on the step
  • After a migration is finished I still see the last step. What, if I potentially want to migrate to another Data Node?

Welcome Step

  • Headline says "Migration to Data node !", remove !
  • In the copy: OpenSearch is written with lower case s
  • In quite some places we say Data node or data node , but on all other pages we spell it Data Node
  • Typos: Migrating to data node require some step the are performed using the UI in this wizard, but it also require some additional step that should be performed on the OS, you current OS/ES cluster and you config files -> Migrating to Data Node requires some steps the are performed using the UI in this wizard, but it also requires some additional steps that should be performed on the OS, your current OS/ES cluster and your config files.
  • You can get more information on the Data node migration [documentation](https://docs.graylog.org/docs/graylog-data-node) missing .
  • In the table, status is written in lower case where all other columns start with an upper case letter

Directory compatibility check

  • OpenSearch is misspelled in some places (Opensearch instead of OpenSearch)
  • Data Node is misspelled in some places

Migration steps

The following points apply to multiple or all of the steps.

  • Data Node is misspelled in some places
  • OpenSearch is misspelled in some places
  • Headline: Rolling upgrade migration., should remove .
  • Text: Follow these steps to migrate your existing migrating an existing OpenSearch 2.x or 1.3.x cluster to Data Node. Missing ., also this sentence seems to miss a word or needs to be restructured.
  • Typo: To start please install Data node on every OS/ES node from you previous setup. , should be To start please install Data node on every OS/ES node from your previous setup.
  • In the table, status is written in lower case where all other columns start with an upper case letter
  • Journal size warning text:
    • Graylog is written in lowercase, I am not sure if it should be uppercase
    • In some places journal is lowercase, in one it's uppercase, we should decide
  • Typo: You can now provision certificate for your data nodes., either a certificate or certificates
  • Typo: Provisionning data nodes certificate., should be Provisioning Data Node certificates
  • Copy:

Please remove the `elasticsearch_hosts` line from you graylog

Ex. `elasticsearch_hosts = https://admin:admin@opensearch1:9200,https://admin:admin@opensearch2:9200,https://admin:admin@opensearch3:9200`

Once the done. Please restart graylog to finish the migration

Should be:

Almost there!

Please remove the `elasticsearch_hosts` line from your `graylog.conf`.

E.g., `elasticsearch_hosts = https://admin:admin@opensearch1:9200,https://admin:admin@opensearch2:9200,https://admin:admin@opensearch3:9200`

Once that's done, please restart Graylog to finish the migration.

import { QUERY_KEY as DATA_NODES_CA_QUERY_KEY } from 'components/datanode/hooks/useDataNodesCA';
import { MIGRATION_STATE_QUERY_KEY } from 'components/datanode/hooks/useMigrationState';

type FormValues = {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should have more specific typing.

{(currentStep.next_steps.length <= 0) && (<p>Please create a certificate renewal policy before proceeding.</p>)}
<MigrationStepTriggerButtonToolbar nextSteps={currentStep.next_steps} onTriggerStep={onTriggerStep} />
</>

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Comment on lines 37 to 38
`;
const StyledSpan = styled.span`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`;
const StyledSpan = styled.span`
`;
const StyledSpan = styled.span`

Comment on lines 31 to 32
}
const Grid = styled.div`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
}
const Grid = styled.div`
}
const Grid = styled.div`

Comment on lines 42 to 43
`;
const StyledPanelGroup = styled(PanelGroup)`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`;
const StyledPanelGroup = styled(PanelGroup)`
`;
const StyledPanelGroup = styled(PanelGroup)`

Comment on lines 29 to 30
</>

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
</>
</>

Comment on lines 39 to 40
);
export default JournalDowntimeWarning;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
);
export default JournalDowntimeWarning;
);
export default JournalDowntimeWarning;

Comment on lines 26 to 27
`;
const JournalSizeWarning = () => (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`;
const JournalSizeWarning = () => (
`;
const JournalSizeWarning = () => (

Comment on lines 31 to 32
</>

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
</>
</>

Comment on lines 28 to 29
`;
const StopMessageProcessing = ({ currentStep, onTriggerStep }: MigrationStepComponentProps) => (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`;
const StopMessageProcessing = ({ currentStep, onTriggerStep }: MigrationStepComponentProps) => (
`;
const StopMessageProcessing = ({ currentStep, onTriggerStep }: MigrationStepComponentProps) => (

@ousmaneo
Copy link
Contributor Author

Thank you @grotlue for the review, here are some remarques about some questions.

  • I can click on non-active steps (previous and next) but nothing happens. Confusingly, the clicked step stays in a visual focused state
  • If I go from Certificate authority to Migration Steps there is no way to get back again

In the current implementation, there is no way to go back once a step is completed; only in the rolling upgrade migration it's is it possible to come back to the Cluster info form.

  • I cannot see an option to cancel the migration process and/or start from step 1 again. After reloading I am still stuck on the step

It's not currently possible to cancel the migration; I already mentioned it to the backend team, and we might add that in the future.

  • After a migration is finished I still see the last step. What, if I potentially want to migrate to another Data Node?

An OpenSearch cluster should only be migrated once. If another data node is added it will join the existing one and form a cluster.

Working on the others.

@ousmaneo ousmaneo force-pushed the datanode-migration-ui branch from 1743e4f to a45d109 Compare February 21, 2024 14:09
@grotlue
Copy link
Contributor

grotlue commented Feb 21, 2024

Here are some additional findings.

I would like to have another look at Remote indexing but due to the errors it's now usable at the moment.

Certificate Renewal Policy Configuration

  • Titles and text are vertically misaligned
    Screenshot 2024-02-21 at 15 58 42

Migration Steps

Remote reindexing

  • Headline: Remote reindexing migration., remove .
  • There are errors every few seconds
    Screenshot 2024-02-21 at 16 01 16

<Row>
<Col md={6}>
<MigrationError errorMessage={currentStep.error_message} />
<Headline>Migration to Data Node !</Headline>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<Headline>Migration to Data Node !</Headline>
<Headline>Migration to Data Node</Headline>

It looks like you updated Graylog and want to configure a Data Node. Data Nodes allow you to index and search through all the messages in your Graylog message database.
</p>
<p>
Using this migration tool you can check the compatibility and follow the steps to migrate your exsisting OpenSearch data to a Data Node.<br />
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Using this migration tool you can check the compatibility and follow the steps to migrate your exsisting OpenSearch data to a Data Node.<br />
Using this migration tool you can check the compatibility and follow the steps to migrate your existing OpenSearch data to a Data Node.<br />


return (
<Col md={6}>
<StyledTitle>Remote reindexing migration.</StyledTitle>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<StyledTitle>Remote reindexing migration.</StyledTitle>
<StyledTitle>Remote reindexing migration</StyledTitle>

return (
<>
<h3>Welcome</h3>
<p>Using the Remote Reindexing will allow you to move the datanode by reindexing the data in your existing cluster to the datanode cluster.</p>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<p>Using the Remote Reindexing will allow you to move the datanode by reindexing the data in your existing cluster to the datanode cluster.</p>
<p>Using the Remote Reindexing will allow you to move the Data Node by reindexing the data in your existing cluster to the Data Node cluster.</p>

<>
<h3>Welcome</h3>
<p>Using the Remote Reindexing will allow you to move the datanode by reindexing the data in your existing cluster to the datanode cluster.</p>
<p>To start please install Data Node on every OS/ES node from you previous setup. You can fing more information on how to download and install the Data Node <DocumentationLink page="graylog-data-node" text="here" />.</p>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<p>To start please install Data Node on every OS/ES node from you previous setup. You can fing more information on how to download and install the Data Node <DocumentationLink page="graylog-data-node" text="here" />.</p>
<p>To start please install Data Node on every OS/ES node from your previous setup. You can find more information on how to download and install the Data Node <DocumentationLink page="graylog-data-node" text="here" />.</p>

return (
<Col md={6}>
<StyledTitle>Remote reindexing migration.</StyledTitle>
<p>Follow these steps to migrate your existing migrating an existing OpenSearch 2.x or 1.3.x cluster to Data
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<p>Follow these steps to migrate your existing migrating an existing OpenSearch 2.x or 1.3.x cluster to Data
<p>Follow these steps to migrate your existing OpenSearch 2.x or 1.3.x cluster to Data

Comment on lines 25 to 31
<p>Almost there !</p>
<p>Please remove the <code>elasticsearch_hosts</code> line from your <code>graylog.conf</code> </p>
<p>Ex. <code>elasticsearch_hosts = https://admin:admin@opensearch1:9200,https://admin:admin@opensearch2:9200,https://admin:admin@opensearch3:9200</code></p>
<Space h="md" />
<p>Once that`&lsquo;`s done. Please restart Graylog to finish the migration.</p>
<MigrationStepTriggerButtonToolbar nextSteps={currentStep.next_steps} onTriggerStep={onTriggerStep} />
</>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<p>Almost there !</p>
<p>Please remove the <code>elasticsearch_hosts</code> line from your <code>graylog.conf</code> </p>
<p>Ex. <code>elasticsearch_hosts = https://admin:admin@opensearch1:9200,https://admin:admin@opensearch2:9200,https://admin:admin@opensearch3:9200</code></p>
<Space h="md" />
<p>Once that`&lsquo;`s done. Please restart Graylog to finish the migration.</p>
<MigrationStepTriggerButtonToolbar nextSteps={currentStep.next_steps} onTriggerStep={onTriggerStep} />
</>
<p>Almost there!</p>
<p>Please remove the <code>elasticsearch_hosts</code> line from your <code>graylog.conf</code> </p>
<p>E.g., <code>elasticsearch_hosts = https://admin:admin@opensearch1:9200,https://admin:admin@opensearch2:9200,https://admin:admin@opensearch3:9200</code></p>
<Space h="md" />
<p>Once that's done, please restart Graylog to finish the migration.</p>
<MigrationStepTriggerButtonToolbar nextSteps={currentStep.next_steps} onTriggerStep={onTriggerStep} />
</>

Comment on lines 25 to 28
<p>Please remove the <code>elasticsearch_hosts</code> line from you Graylog configuration file (<code>graylog.conf</code>). </p>
<p>Ex. <code>elasticsearch_hosts = https://admin:admin@opensearch1:9200,https://admin:admin@opensearch2:9200,https://admin:admin@opensearch3:9200</code></p>
<Space h="md" />
<p>Once that is done please proceed to the next step.</p>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<p>Please remove the <code>elasticsearch_hosts</code> line from you Graylog configuration file (<code>graylog.conf</code>). </p>
<p>Ex. <code>elasticsearch_hosts = https://admin:admin@opensearch1:9200,https://admin:admin@opensearch2:9200,https://admin:admin@opensearch3:9200</code></p>
<Space h="md" />
<p>Once that is done please proceed to the next step.</p>
<p>Please remove the <code>elasticsearch_hosts</code> line from your Graylog configuration file (<code>graylog.conf</code>).</p>
<p>E.g., <code>elasticsearch_hosts = https://admin:admin@opensearch1:9200,https://admin:admin@opensearch2:9200,https://admin:admin@opensearch3:9200</code></p>
<Space h="md" />
<p>Once that is done please proceed to the next step.</p>

const ShutdownClusterStep = ({ currentStep, onTriggerStep }: MigrationStepComponentProps) => (
<>
<p>The migration from your current <code>OpenSearch</code> to the Data Node is almost done.</p>
<p>to finish please shut down your <code>OpenSearch</code> cluster before continuing.</p>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<p>to finish please shut down your <code>OpenSearch</code> cluster before continuing.</p>
<p>To finish please shut down your <code>OpenSearch</code> cluster before continuing.</p>

<>
<h3>Welcome</h3>
<p>Using the Remote Reindexing will allow you to move the Data Node by reindexing the data in your existing cluster to the Data Node cluster.</p>
<p>To start please install Data Node on every OS/ES node from your previous setup. You can find more information on how to download and install the Data Node <DocumentationLink page="graylog-data-node" text="here" />.</p>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<p>To start please install Data Node on every OS/ES node from your previous setup. You can find more information on how to download and install the Data Node <DocumentationLink page="graylog-data-node" text="here" />.</p>
<p>To start please install Data Node on every OS/ES node from your previous setup. You can find more information on how to download and install the Data Node <DocumentationLink page="graylog-data-node" text="here" />.</p>

@ousmaneo ousmaneo requested a review from grotlue February 22, 2024 07:51
<p>
Using this migration tool you can check the compatibility and follow the steps to migrate your existing OpenSearch data to a Data Node.<br />
</p>
<p>Migrating to Data Node requires some steps the are performed using the UI in this wizard, but it also requires some additional steps that should be performed on the OS, your current OS/ES cluster and your config files.</p>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<p>Migrating to Data Node requires some steps the are performed using the UI in this wizard, but it also requires some additional steps that should be performed on the OS, your current OS/ES cluster and your config files.</p>
<p>Migrating to Data Node requires some steps that are performed using the UI in this wizard, but it also requires some additional steps that should be performed on the OS, your current OS/ES cluster and your config files.</p>

<p>Please remove the <code>elasticsearch_hosts</code> line from your <code>graylog.conf</code></p>
<p>E.g., <code>elasticsearch_hosts = https://admin:admin@opensearch1:9200,https://admin:admin@opensearch2:9200,https://admin:admin@opensearch3:9200</code></p>
<Space h="md" />
<p>Once that`s done, please restart Graylog to finish the migration.</p>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<p>Once that`s done, please restart Graylog to finish the migration.</p>
<p>Once that's done, please restart Graylog to finish the migration.</p>

@gally47 gally47 requested a review from grotlue February 22, 2024 09:26
@gally47 gally47 merged commit 7d6b291 into master Feb 22, 2024
6 checks passed
@gally47 gally47 deleted the datanode-migration-ui branch February 22, 2024 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Migration Page UI
5 participants